Learning to Lemmatise Slovene Words
Identifieur interne : 000319 ( Main/Exploration ); précédent : 000318; suivant : 000320Learning to Lemmatise Slovene Words
Auteurs : Sašo Džeroski [Slovénie] ; Tomaž Erjavec [Slovénie]Source :
- Lecture Notes in Computer Science [ 0302-9743 ] ; 2000.
Abstract
Abstract: Automatic lemmatisation is a core application for many language processing tasks. In inflectionally rich languages, such as Slovene, assigning the correct lemma to each word in a running text is not trivial: nouns and adjectives, for instance, inflect for number and case, with a complex configuration of endings and stem modifications. The problem is especially difficult for unknown words, as word forms cannot be matched against a lexicon giving the correct lemma, its part-of-speech and paradigm class. The paper discusses a machine learning approach to the automatic lemmatisation of unknown words, in particular nouns and adjectives, in Slovene texts. We decompose the problem of learning to perform lemmatisation into two subproblems: the first is to learn to perform morphosyntactic tagging, and the second is to learn to perform morphological analysis, which produces the lemma from the word form given the correct morphosyntactic tag. A statistics-based trigram tagger is used to learn to perform morphosyntactic tagging and a first-order decision list learning system is used to learn rules for morphological analysis. The dataset used is the 90.000 word Slovene translation of Orwell’s ‘1984’, split into a training and validation set. The validation set is the Appendix of the novel, on which extensive testing of the two components, singly and in combination, is performed. The trained model is then used on an open-domain testing set, which has 25.000 words, pre-annotated with their word lemmas. Here 13.000 nouns or adjective tokens are previously unseen cases. Tested on these unknown words, our method achieves an accuracy of 81% on the lemmatisation task.
Url:
DOI: 10.1007/3-540-40030-3_5
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 000431
- to stream Istex, to step Curation: 000431
- to stream Istex, to step Checkpoint: 000267
- to stream Main, to step Merge: 000345
- to stream Main, to step Curation: 000319
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct:series"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Learning to Lemmatise Slovene Words</title><author><name sortKey="Dzeroski, Saso" sort="Dzeroski, Saso" uniqKey="Dzeroski S" first="Sašo" last="Džeroski">Sašo Džeroski</name></author><author><name sortKey="Erjavec, Tomaz" sort="Erjavec, Tomaz" uniqKey="Erjavec T" first="Tomaž" last="Erjavec">Tomaž Erjavec</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:927620C06EAB28B7791C8F925DB80BB0EEA4CFAB</idno><date when="2000" year="2000">2000</date><idno type="doi">10.1007/3-540-40030-3_5</idno><idno type="url">https://api.istex.fr/document/927620C06EAB28B7791C8F925DB80BB0EEA4CFAB/fulltext/pdf</idno><idno type="wicri:Area/Istex/Corpus">000431</idno><idno type="wicri:Area/Istex/Curation">000431</idno><idno type="wicri:Area/Istex/Checkpoint">000267</idno><idno type="wicri:explorRef" wicri:stream="Istex" wicri:step="Checkpoint">000267</idno><idno type="wicri:doubleKey">0302-9743:2000:Dzeroski S:learning:to:lemmatise</idno><idno type="wicri:Area/Main/Merge">000345</idno><idno type="wicri:Area/Main/Curation">000319</idno><idno type="wicri:Area/Main/Exploration">000319</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Learning to Lemmatise Slovene Words</title><author><name sortKey="Dzeroski, Saso" sort="Dzeroski, Saso" uniqKey="Dzeroski S" first="Sašo" last="Džeroski">Sašo Džeroski</name><affiliation wicri:level="1"><country xml:lang="fr">Slovénie</country><wicri:regionArea>Department for Intelligent Systems, Jožef Stefan Institute, Jamova 39, SI-1000, Ljubljana</wicri:regionArea><wicri:noRegion>Ljubljana</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Slovénie</country></affiliation></author><author><name sortKey="Erjavec, Tomaz" sort="Erjavec, Tomaz" uniqKey="Erjavec T" first="Tomaž" last="Erjavec">Tomaž Erjavec</name><affiliation wicri:level="1"><country xml:lang="fr">Slovénie</country><wicri:regionArea>Department for Intelligent Systems, Jožef Stefan Institute, Jamova 39, SI-1000, Ljubljana</wicri:regionArea><wicri:noRegion>Ljubljana</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Slovénie</country></affiliation></author></analytic><monogr></monogr><series><title level="s">Lecture Notes in Computer Science</title><imprint><date>2000</date></imprint><idno type="ISSN">0302-9743</idno><idno type="ISSN">0302-9743</idno></series><idno type="istex">927620C06EAB28B7791C8F925DB80BB0EEA4CFAB</idno><idno type="DOI">10.1007/3-540-40030-3_5</idno><idno type="ChapterID">5</idno><idno type="ChapterID">Chap5</idno></biblStruct></sourceDesc><seriesStmt><idno type="ISSN">0302-9743</idno></seriesStmt></fileDesc><profileDesc><textClass></textClass><langUsage><language ident="en">en</language></langUsage></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: Automatic lemmatisation is a core application for many language processing tasks. In inflectionally rich languages, such as Slovene, assigning the correct lemma to each word in a running text is not trivial: nouns and adjectives, for instance, inflect for number and case, with a complex configuration of endings and stem modifications. The problem is especially difficult for unknown words, as word forms cannot be matched against a lexicon giving the correct lemma, its part-of-speech and paradigm class. The paper discusses a machine learning approach to the automatic lemmatisation of unknown words, in particular nouns and adjectives, in Slovene texts. We decompose the problem of learning to perform lemmatisation into two subproblems: the first is to learn to perform morphosyntactic tagging, and the second is to learn to perform morphological analysis, which produces the lemma from the word form given the correct morphosyntactic tag. A statistics-based trigram tagger is used to learn to perform morphosyntactic tagging and a first-order decision list learning system is used to learn rules for morphological analysis. The dataset used is the 90.000 word Slovene translation of Orwell’s ‘1984’, split into a training and validation set. The validation set is the Appendix of the novel, on which extensive testing of the two components, singly and in combination, is performed. The trained model is then used on an open-domain testing set, which has 25.000 words, pre-annotated with their word lemmas. Here 13.000 nouns or adjective tokens are previously unseen cases. Tested on these unknown words, our method achieves an accuracy of 81% on the lemmatisation task.</div></front></TEI><affiliations><list><country><li>Slovénie</li></country></list><tree><country name="Slovénie"><noRegion><name sortKey="Dzeroski, Saso" sort="Dzeroski, Saso" uniqKey="Dzeroski S" first="Sašo" last="Džeroski">Sašo Džeroski</name></noRegion><name sortKey="Dzeroski, Saso" sort="Dzeroski, Saso" uniqKey="Dzeroski S" first="Sašo" last="Džeroski">Sašo Džeroski</name><name sortKey="Erjavec, Tomaz" sort="Erjavec, Tomaz" uniqKey="Erjavec T" first="Tomaž" last="Erjavec">Tomaž Erjavec</name><name sortKey="Erjavec, Tomaz" sort="Erjavec, Tomaz" uniqKey="Erjavec T" first="Tomaž" last="Erjavec">Tomaž Erjavec</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Ticri/explor/TeiVM2/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000319 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000319 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Ticri
|area= TeiVM2
|flux= Main
|étape= Exploration
|type= RBID
|clé= ISTEX:927620C06EAB28B7791C8F925DB80BB0EEA4CFAB
|texte= Learning to Lemmatise Slovene Words
}}
| This area was generated with Dilib version V0.6.31. |  |